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ABSTRACT 

In  a previous  paper,  I described  a technique  for  designing  an  observer  test  in  an  iterative 
manner.  In  field  tests  to  compare  the  observability  of  combat  vehicles,  the  test  designer  must 
select  the  optimum  number  of  observation  opportunities  in  order  to  balance  collecting  enough  data 
to  draw  valid  conclusions  against  the  high  cost  of  supporting  vehicles  and  personnel  at  a test  site. 
The  test  designer  can  select  the  number  of  observations,  N,  so  that  a given  experimental  difference 
in  detectability  will  be  statistically  significant  at  a given  confidence  level.  Alternatively,  the  test 
designer  can  select  N so  that  the  probability  of  rejecting  a given  underlying  difference  in 
detectability  is  less  than  a given  amount.  The  test  designer,  however,  generally  lacks  key 
parameters  for  the  efficient  design  of  the  test.  Namely,  the  designer  lacks  the  detection 
probabilities  of  the  vehicles  at  each  range.  The  standard  deviation  of  the  difference  in  detection 
probability  depends  upon  the  detection  probability  itself.  Therefore,  the  test  designer  must  select 
the  number  of  observations  for  each  range  based  upon  the  conservative  assumption  that  the 
probabilities  are  near  50%,  the  probability  for  the  maximum  standard  deviation.  In  the  previous 
paper,  an  iterative  technique  of  test  design  was  described.  In  this  technique,  the  test  designer 
modifies  the  test  matrix  as  the  test  progresses.  Early  test  results  yield  estimates  of  the  probability 
of  detection  for  each  vehicle  at  each  range.  Based  on  these  estimates,  the  test  designer  reallocates 
the  number  of  observations  among  the  ranges,  improving  the  efficiency  of  the  test. 

In  this  paper,  I present  an  algorithm  to  implement  this  iterative  technique.  Initially, 
observation  opportunities  are  equally  distributed  among  the  ranges  of  vehicle  sites.  But,  as  the  test 
progresses,  the  observation  opportunities  are  redistributed  among  the  ranges  according  to 
estimates  of  detection  probability'.  For  example,  the  fraction  of  opportunities  at  the  range  near  50% 
probability  is  increased  while  the  fraction  near  10%  probability  is  decreased.  The  algorithm 
handles  the  special  cases  that  occur  when  the  estimated  probability  is  I or  0. 


1.  THE  EXPERIMENTAL  SITUATION 

Figure  1 illustrates  a typical  test  setup  for  a test  of  observability.  Observers  are  stationed  at  a 
fixed  site  and  attempt  to  detect  a vehicle  in  their  field  of  view.  For  each  observation  opportunity, 
the  test  personnel  record  the  number  of  detections.  In  analyzing  the  data,  the  analyst  groups  the 
observations  into  range  bins  and  compares  the  proportion  of  detections  for  each  test  vehicle. 

For  such  a field  test,  the  test  designer  must  select  the  optimum  number  of  observation 
opportunities  at  each  range.  The  designer  must  balance  collecting  enough  data  to  draw  valid 
conclusions  against  the  high  cost  of  supporting  vehicles  and  personnel  at  a test  site. 
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2.  DESIGNING  THE  EXPERIMENT 


The  test  designer  selects  the  number  of  observations  in  order  to  achieve  two  objectives.  First,  the 
number  of  observations  must  be  large  enough  so  that  the  probabilities  of  errors  are  below  the 
desired  level.  And,  second,  the  number  of  observations  must  be  as  small  as  possible  to  minimize 
testing  time  and  cost. 

Table  I illustrates  the  types  of  errors  that  can  occur  in  drawing  conclusions  from  an  observation 
test.  The  goal  of  an  observation  test  is  to  judge  whether  the  observed  difference  in  detection 
probability,  Pd,  is  unusual  enough  to  reject  the  null  hypothesis  that  Vehicle  A and  Vehicle  B 
have  the  same  Pd.  If  the  analyst  concludes  that  the  null  hypothesis  cannot  be  rejected,  then  either 
the  analyst  has  made  a correct  decision  or  a Type  II  error.  On  the  other  hand,  a decision  to  reject 
the  null  hypothesis  will  be  either  correct  or  a Type  I error.  In  terms  of  countermeasure 
effectiveness,  a Type  I error  is  an  erroneous  conclusion  than  an  ineffective  countermeasure  is 
effective,  while  a Type  II  error  miss  judges  an  effective  countermeasure. 

In  a previous  paper  [1],  I discussed  the  number  of  observation  opportunities  required  to  control 
the  probability  of  these  errors.  For  the  case  of  an  average  Pd  of  0.5,  Figure  2 plots  the  number  of 
observations  required  to  maintain  the  probability  of  Type  I and  Type  II  errors  at  less  than  5%. 
For  example,  85  observations  of  each  vehicle  are  adequate  for  an  observed  0.15  difference  to  be 
significant  with  less  than  5%  chance  of  a Type  I error.  On  the  other  hand,  for  less  than  5% 
chance  of  committing  a Type  II  error  when  the  underlying  probabilities  differ  by  0.15,  the 
requirement  is  3 1 6 observation  opportunities  per  vehicle. 

3.  AN  ITERATIVE  TECHNIQUE  FOR  TEST  DESIGN 


An  iterative  technique  can  reduce  the  number  of  observation  opportunities  required  for  a test. 
Observation  data  is  described  by  the  binary  distribution.  But  the  binary  distribution  has  the 
property  that  the  standard  deviation  depends  upon  the  probability.  If  p is  the  probability  and  N is 
the  number  of  trials,  then  the  standard  deviation  of  the  number  of  successes,  o,  is  given  by 

*%et  ecliom  = j NP(l  - P)  (1) 

And  the  standard  deviation  of  the  proportion  of  successes  is  given  by 


<7 


proportion 


det  ec  lions 


N 


IpCI-p) 

i N 


(2) 


Figure  3 plots  Equation  2 for  an  N of  100,  200  and  300.  Figure  4 plots  the  same  equation 
normalized  to  its  maximum  at  p = 0.5. 


Figure  5 further  shows  how  the  number  of  trials  changes  with  probability  if  the  standard 
deviation  is  held  fixed.  For  example,  in  comparison  to  a probability  of  0.5,  only  60%  as  many 
trials  are  need  at  a probability  of  0. 1 . 
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To  use  this  property  of  the  binary  distribution,  the  test  designer  modifies  the  test  matrix  as  the 
test  progresses.  Early  test  results  yield  estimates  of  the  probability  of  detection  for  each  vehicle 
at  each  range.  Based  on  these  estimates,  the  test  designer  reallocates  the  number  of  observations 
among  the  ranges,  improving  the  efficiency  of  the  test  design.  Figure  5 illustrates  a sample  of  the 
improvement  in  efficiency  that  this  technique  can  achieve.  Initially  the  test  designer  would  have 
selected  the  number  of  observations  at  each  test  range  by  assuming  the  worst-case  value  of  Pd  = 
0.5.  But  as  the  test  progresses,  the  designer  would  use  estimates  of  Pd  from  the  early  measures  to 
redesign  the  test.  For  example,  at  range  20,  the  designer  would  reduce  the  number  of 
observations  from  314  to  251.  Overall  in  this  sample  test,  the  test  designer  would  reduce  the 
number  of  observations  from  1570  to  1193,  a reduction  of  24%. 


4.  IMPLEMENTATION  OF  THE  ITERATIVE  TECHNIQUE  FOR  TEST  DESIGN 

A.  Algorithm  for  Fixed  Number  of  Observation  Opportunities 

The  first  step  is  selection  of  the  number  of  observation  opportunities  for  each  range  based  on  the 
conservative  assumption  of  a Pd  of  50%  at  each  range.  From  figures  x and  y,  the  designer  selects 
No  to  reduce  the  probabilities  of  Type  I and  II  errors  to  acceptable  levels. 


Next,  the  designer  calculates  estimates  of  P'd , the  average  probability  of  detection  at  range  Rj, 
based  on  the  results  of  the  first  day  of  observations,  as  follows: 


Da  + 
Na  +N 


where  Da  and  Db  are  the  numbers  of  detections  of  Vehicles  A and  B and  NA  and  Nb  are  the 
number  of  observation  opportunities  for  Vehicles  A and  B. 


To  account  for  special  cases  when  there  are  no  detections  or  100%  detections,  Equation  3 should 
be  modified  to  prevent  later  setting  the  number  of  observation  opportunities  to  zero.  The 
following  form  covers  these  special  cases. 


If  (Da+Db)/(Na+Nb)  < 0. 1 , then  P'  = 0. 1 
If  (Da+Db)/(Na+Nb)  > 0.9,  then  J»J  = 0.9 

Otherwise.  PI  = — -+— g 


(4) 


Next,  beginning  with  Equation  3,  the  designer  calculates  N;,  the  new  number  of  observation 
opportunities  for  range  Rj: 


< x 


N. 


(5) 
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Note  that  a is  the  same  for  all  ranges.  Solving  for  Ni  yields, 


N, 


p:,{ i-pj) 


(6) 


And,  keeping  the  total  number  of  observation  opportunities  constant  at  NT  , then 


N, 


=±N'±rt-r;) 


(7) 


Solving  for  a2,  gives 
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Now  the  test  continues  with  the  modified  number  of  opportunities  for  observation  at  each  range. 
AUhe  endtf  e”  eeding  day,  the  designer  can  recalcuiate  the  N,'s  in  the  satne  way  to  refute 

the  design. 


B.  Algorithm  for  Minimal  Number  of  Observation  Opportunities 

new  number  of  observations  can  be  calculated  by  using  Equation  2 to  keep  the. stand 
deviations  constant, 
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Solving  for  N ”e"'  gives, 
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Again,  at 
design. 


the  end  of  each  succeeding  day,  the  designer  can  recalculate  the  N,’s  to  refine  the 
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5.  CONCLUSION 


In  this  paper,  I have  presented  an  iterative  technique  of  using  early  test  results  in  an  observation 
test  to  improve  the  overall  efficiency  of  the  test  design 
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Possible  Outcomes  of  Hypothesis  Testing 


Is  Vehicle  A Less  Detectable 
Than  Vehicle  B? 


Decision: 

No 

Yes 

Accept  Null  Hypothesis 

Correct 

I ype  !s  Error 

Accept  Alternative  Hypothesis 

T ype  1 Error 

Correct 

Table  1.  Definitions  of  Type  I and  Type  II  errors. 
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vers. 


Fixed  Observers 


Figure  1.  Experimental  setup  for  an  observer  test. 


Number  of  Opportunities 
i to  Meet  Test  Criteria 


Pd  Difference 


Figure  2.  Number  of  observation  opportunities  required  to  meet  test  criteria. 
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Standard  Deviation  vs.  Probability 


Probability 


Figure  3.  Standard  deviation  of  a binary  distribution  for  100,  200  and  300  trials. 


Number  of  Observation  for  Constant  Standard  Deviation 

N (%  of  N for  0.5) 


Initial  and  Optimum  Number  of  Observations  per  Vehicle 


Figure  5.  An  example  of  iterative  design  of  an  observer  test. 
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